Repository navigation
Account for every region that drops the GIL, name a thread count in each, and explain the fork hang - #10
Merged
Conversation
`build/` was spelled into the Makefile fifteen times, so there was nowhere to put a second build. That matters for a sanitiser: `build/libbeam.a` and the suites beside it are what `verification/check.py` reads, rebuilding them costs about an hour on a loaded box, and an instrumented archive left at that path would be handed to the gate as though it were the shipping library. `BUILD` is now a variable, `make clean` honours it, and `make tsan` sets it to `build-tsan`. `SANFLAGS` is appended to both the compile and the link rather than replacing CFLAGS, because a sanitiser build that restates the flags is how `-mpopcnt` or `-Isrc/native/internal` goes missing from one of them -- and that failure arrives as a wrong answer rather than as a compile error. The optimisation level is deliberately not lowered either: TSan instruments memory accesses, which survive -O2, and an instrumented build that tests different code from the one that ships answers a question nobody asked. `make tsan` requires clang, libomp and Archer, and refuses rather than falling back to GCC. That is not a preference. libgomp carries no sanitiser annotations -- `nm -D libgomp.so.1` finds no `__tsan_*` and no `Annotate*` symbol -- so TSan sees no happens-before edge between any two OpenMP threads, every implicit barrier is invisible, and every region that writes a result somebody later reads is reported as a race. Measured on `test_distance` alone at 8 threads: 1186 reports. 1184 were main-thread-against-worker. The other two were worker-against-worker and were also barrier artifacts, between pool threads reused across different regions -- `advance._omp_fn.0` against `advance._omp_fn.2`. There is no filter that separates a real intra-region race from a barrier-separated one when the tool has no edges at all, so none is shipped here, and the target fails with the apt line that fixes it instead of producing a log that cannot be read. Two suites stay out of the instrumented set: `test_model` and `test_expected_counts_accounting` link with `-Wl,--wrap=malloc` while TSan replaces the allocator itself, so the wrapper and the sanitiser contend for one symbol. Both test allocation failure rather than parallelism. `tests-only` builds every suite without running one, which neither `test` nor `list-tests` does. The wrap-link guard read the literal target `build/tests/native/<suite>` out of the Makefile and now reads `$(BUILD)/tests/native/<suite>`. The invariant it checks is unchanged; only the spelling moved.
On a clean tree the wiring check printed
OK thread plan wiring (29 analytics regions planned, no escapes)
A reader takes that for a statement about the library. It was a statement
about two files, `analytics_bindings.c` and `common.c`, named in a
`PLANNED_BINDINGS` tuple. There are fourteen binding files, and the other
twelve hold 40 regions that drop the GIL and resolve no thread plan, so
`beam_threads()` falls through to `threads_cap()` -- with no cap set, that is
`omp_get_max_threads()`: one thread per core, the default plan.h exists to
stop asking for. On this 16-core box with 15 free, `BEAM.parallel_plan()`
reports a plan of 8 while `effective_threads()` is 16, so an analytics call
runs 8 and `py_observe` runs 16. On the saturated 48-core node it asks for 48,
which the design notes measure at 301x worse than serial at half that.
`model_bindings.c`'s 14 are the expensive half of the library: observe,
finalize, build, load, save, temper, posterior, set_prior, subtract, add and
four introspection calls.
That is a deferral and not a defect -- decision 9 reads "Query-time calls
only, for now; fitting keeps today's behaviour until it is profiled" -- so the
40 are recorded rather than failed. Recording buys what a printed count
cannot: a NEW unplanned region fails the gate, so the deferral cannot quietly
grow, and wiring one up prints a note asking to be re-recorded. The lesson
behind that is `c_symbols.txt`, a baseline with no recorder at all, which
drifted by thirty symbols across four merges. Regions are identified by
function rather than by line, because a line-numbered baseline churns on every
unrelated edit above it and teaches its reader to re-record without reading.
The count is 40 and not 41. The scan is a text search, and
`mixture_bindings.c` opens with a doc comment explaining what holds across
"the `Py_BEGIN_ALLOW_THREADS` region". That sentence counted as a 41st region,
attributed to no function at all because it sits above the first one -- the
same mistake `public_text_names` already records in this file, where
`jet_sweep` matched a prose comment and was counted as public API. Comments
are now stripped, tracking `/* */` across lines, before anything is matched.
Five mutations against the real gate, each caught by the intended branch: a
new unplanned region fails; a planned region that stops planning fails; a
properly wired region prints a note and does not fail; an unbalanced
BEGIN/END fails naming its own function; a clean tree passes. The last of
those is why `unbalanced` is returned separately -- the first attempt left the
scanner inside a region for the rest of the file, so every later `return`
became an escape, and the mutation printed 87 escape lines without ever
saying the macro was unbalanced.
A missing baseline fails the gate unless this is the run recording it, the
same exemption and the same bound as `math_values`.
An OpenMP thread pool does not survive `fork`. Only the thread that called it
exists in the child, so the child waits at its first parallel computation on
workers that were never copied: no exception, no traceback, no exit. That is
the worst failure mode a library has, and the documentation that says to use
spawn gets read after the hang rather than before it.
So the child says so on the way past. The condition is "this process had more
than one native thread when it forked", read from `/proc/self/status` because
an OpenMP worker is not a Python thread and `threading.active_count()` cannot
see it. Verified both ways: a fresh import reports one thread, and building a
model under `set_threads(4)` reports four, so the predicate is off for a fork
that is safe and on for the one that is not.
The condition is deliberately broader than "BEAM had a pool" -- another
library's threads satisfy it too, and this will occasionally warn a child that
would have been fine. Forking a multithreaded process is the hazard whoever
started the threads, the warning is suppressible by category, and a guard that
stays quiet through the case it exists for is worth less than one that is
sometimes early. An unreadable thread count reports one rather than guessing
high, which makes the guard silent on a platform without /proc instead of
warning on every fork there.
Four tests, and one of them was wrong until it was mutation-tested.
`test_a_fork_with_nothing_running_is_silent` used `simplefilter("error")`,
which `_after_fork_in_child` swallows on purpose, so it passed against a guard
mutated to warn on every fork; it now asserts on the recorded warning list.
Removing the `os.register_at_fork` call is caught by the end-to-end fork test
and by none of the other three, which is why an end-to-end fork test is there
at all. Dropping "spawn" from the message, and guessing high on an unreadable
count, are each caught by their own test.
The reference page recommended spawn without saying what happens otherwise,
so the recommendation had nothing behind it. It now states the consequence,
and that a fork before any computation is safe.
Every name added is underscore-private, so BEAM's declared surface is
unchanged: `python_api.txt` is byte-identical at 381 signature lines.
Decision 9 left fitting on the old behaviour "until it is profiled". This is
that profile, and the wiring it supports. Matched-sample sweep, five processes
per thread count, median of medians, 16-core desktop at load 2.8-6.5:
order 2 order 3
1 thr 189.0 ms 1.00x 630.8 ms 1.00x
3 thr 159.7 ms 1.18x 405.6 ms 1.56x
6 thr 169.1 ms 1.12x 378.7 ms 1.67x
8 thr 171.2 ms 1.10x 397.7 ms 1.59x
12 thr 215.2 ms 0.88x 509.9 ms 1.24x
16 thr 410.7 ms 0.46x 693.4 ms 0.91x
Fitting parallelises weakly and degrades past eight. The behaviour decision 9
preserved asked `omp_get_max_threads()`, which on this box is sixteen -- the
worst point on both curves, and 2.2x worse than serial at order 2. Measured
again in auto mode after wiring: order 2 410.7 -> 178.1 ms, order 3
693.4 -> 405.2 ms, with the plan reporting eight.
A fitting-specific ceiling of four to six was considered and rejected. It buys
about 7% at order 2 and 5% at order 3 over the existing ceiling, which does
not pay for a second measured constant that has to stay correct on machines
nobody has measured. The ceiling is a cap, and both workload classes now
measured -- analytics, which scales to eight, and fitting, which peaks at
three to six -- are better served by any value at or below eight than by one
thread per core.
The other 26 of the 40 are query-time calls that decision 9's own rule already
covered and that Task 4 did not reach: mixture, query, publicness, scoring,
distance, fingerprint, insights, sample and top_k. The analytics ceiling was
measured on that workload class, so they need no separate profile.
Mechanical, and checked before any edit: all 40 regions sit in files that
already include python_bindings.h, none contains a `return` or `goto`, and
none lacks a matching Py_END_ALLOW_THREADS. The macro is a drop-in. The
replacement was made comment-aware so that mixture_bindings.c's doc comment
was not silently rewritten; it is corrected by hand instead, because the
region it describes is no longer spelled that way.
No recorded number moved, across 40 entry points whose thread count changed:
78 numeric keys, 58 math values bit for bit, the saved-model sha256, thread
independence at 1/4/16 and 416 quantities against enumeration all compared
equal. That is the reduction design earning its keep -- no floating-point sum
reduction anywhere, and the sums that exist split into a fixed 256 chunks
summed in index order. The extension's hash did change, since this is the
first commit in the series to edit C.
The baseline is kept although nothing is exempt any more: a region added later
without the macro is a failure against it, which is why this check stopped
being a printed number. Its line now reads "all 69 ... resolve a thread plan",
phrased without the word "unplanned" -- "0 unplanned" is a sentence a reader
parses twice to learn that nothing is wrong.
Not answered here: why fitting degrades at all. A 2.2x penalty at sixteen
threads points at barrier or contention cost in the per-length regions rather
than at a thread count being wrong, which would be an optimisation and
possibly a larger one. And this is one idle desktop; the loaded 48-core node,
where auto mode asked for 48, still needs its own run.
…planned
A review of this branch found the gate certifying something it does not
check. It printed
OK thread plan wiring (all 69 GIL-releasing regions resolve a thread plan)
under the heading "The thread plan is actually in force". The macro scan
proves a plan is RESOLVED at the Python boundary; it cannot prove the region
underneath NAMES it. One did not:
`src/native/occupancy/poisson_binomial/pgf.c` carried no `num_threads` clause
at all, the only such region in src/native. `py_publicness_pgf` is one of the
entry points wrapped three commits ago and the wrap did nothing there --
measured at sixteen native threads after `set_threads(1)`. Wrapping an entry
point and applying a plan are different things: a plan is applied by naming
it in the clause.
That is the same fault as the thirty drifting symbols and the 372-difference
API check, reproduced while fixing them: a green line that asserts more than
it verifies. plan.h carried the matching overstatement -- "Every `omp
parallel` in the implementation carries `num_threads(...)` with this" -- which
was untrue twice over, since 29 regions name a budget-derived local rather
than `beam_threads()` and one named nothing.
`check_lib.thread_count_clauses` is the check the parallel policy plan
specified as Task 5 and that was never built. It joins backslash
continuations, strips comments, finds every `#pragma omp parallel` and
paren-matches the `num_threads(...)` expression. Matched rather than
pattern-captured on purpose: `conc ? (int)conc : 1` contains parentheses, and
a regex stopping at the first `)` reads it as `conc ? (int` and calls it
forbidden. The plan asked for brace matching for the same reason, having
already produced one wrong finding with a grep.
It fails on a region naming no count and on a count outside
`_PERMITTED_THREAD_COUNTS`. That set is a tripwire in source rather than a
recorded baseline: adding a name is a decision someone makes in review, and
the question it forces is what resolves that count. The seven in the tree are
`beam_threads()` (64), `conc` (14), `nth` (10), `team` (2), `conc_e`, `conc_z`
and `conc ? conc : 1`. What it exists to reject is `omp_get_max_threads()`,
`omp_get_num_procs()` and any literal -- each of which asks for one thread per
core behind the caller's back.
`#pragma omp for` and `omp critical` are deliberately not matched: an `omp
for` inside an enclosing `parallel` takes no `num_threads`, and requiring one
would be a demand the language rejects.
Both failure paths were mutated against the real gate, not only unit-tested.
Stripping the clause from math/moments.c gives "1 parallel region(s) name no
thread count"; changing it to `omp_get_max_threads()` gives "1 region(s) name
a thread count that is not a permitted form". Eleven unit tests cover the
parsing rules, including the four forbidden forms above.
Two build faults from the same review. `make test` ran `tools/qa/run c` with
no `--bin-dir`, so `make test BUILD=elsewhere` linked into elsewhere/ and then
ran build/, reporting a pass for binaries it had not built. And build-tsan/
was not ignored, so `make tsan` left the tree dirty and `make clean` did not
remove it.
No recorded number moved, although this edits C and the extension's hash did.
The same review found the guard added two commits ago mute in the one configuration that asks to hear about warnings most loudly. Under `-W error`, or a pytest `filterwarnings = error`, `warnings.warn` RAISES rather than printing -- and the `except Exception: pass` that existed so a child which cannot warn still runs caught that and threw it away. Measured: with warnings as errors, a child forked after a BEAM computation hung and printed nothing at all. There is now an `os.write(2, ...)` fallback of a constant built at import, which is the most that can safely be done in a child forked from a threaded process, where anything that allocates or takes a lock may not come back. It degrades to a stderr line rather than an exception on purpose: an exception raised from an at-fork handler has nowhere to go. The new test uses `capfd` rather than `capsys`, since the fallback writes to the file descriptor and never passes through `sys.stderr`, and it fails when the fallback is removed. The message also stated a hang as fact on a deliberately broad predicate. The condition is "this process had more than one native thread", which a Jupyter kernel or another library's pool satisfies with BEAM never having run, so "the first parallel computation in this child will hang" was wrong for those children. It now says an OpenMP pool does not survive fork and that IF this child uses BEAM its first parallel computation MAY hang. The reference page said "A fork before any computation has run is safe, and is not warned about", which was false for any multithreaded process. It now says what the check actually is, that another library's threads can trigger it, that the category can be filtered, and that a fork from a single-threaded process is not warned about. `gil_regions` was annotated as returning a 2-tuple and has returned a 3-tuple since the unbalanced-region branch was added; the docstring was updated at the time and the annotation was not. pytest_cases 1242 -> 1243, one id added and none removed. No other baseline moved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six pieces of the parallelism subsystem, each green on its own commit. Commits 5 and 6 answer an independent review of 1-4.
1. An instrumented build that cannot overwrite the real one
build/was spelled into the Makefile fifteen times, so there was nowhere to put a second build — andbuild/libbeam.ais whatverification/check.pyreads.BUILDis now a variable,make cleanhonours it, andmake tsansets it tobuild-tsan. Addstests-only, which builds every suite without running one.make tsanrequires clang, libomp and Archer, and refuses rather than falling back to GCC. That is a measured decision, recorded in the Makefile so nobody repeats the experiment:libgompcarries no sanitiser annotations (nm -D libgomp.so.1finds no__tsan_*, noAnnotate*), so TSan sees no happens-before edge between any two OpenMP threads and every implicit barrier is invisible. Ontest_distancealone at 8 threads: 1186 reports, 1184 main-thread-against-worker, and the other two were also barrier artifacts between pool threads reused across different regions (advance._omp_fn.0againstadvance._omp_fn.2). No filter separates a real intra-region race from a barrier-separated one when the tool has no edges at all, so none is shipped.2. The wiring check counts all fourteen binding files
On a clean tree it used to print
thread plan wiring (29 analytics regions planned, no escapes)— which reads as a statement about the library and was a statement about two files named in aPLANNED_BINDINGStuple. The other twelve hold 40 regions that drop the GIL and resolve no thread plan, sobeam_threads()falls through toomp_get_max_threads(): one thread per core. On this 16-core box with 15 free,parallel_plan()reports 8 whileeffective_threads()is 16 — an analytics call runs 8 andpy_observeruns 16. On the saturated 48-core node it asks for 48.That is a deferral, not a defect (decision 9: "Query-time calls only, for now; fitting keeps today's behaviour until it is profiled"), so the 40 are recorded rather than failed. Recording buys what a printed count cannot: a new unplanned region fails the gate, so the deferral cannot quietly grow, and wiring one up prints a note asking to be re-recorded. The lesson behind that is
c_symbols.txt, which had no recorder and drifted by thirty symbols across four merges.New line:
OK thread plan wiring (29 of 69 GIL-releasing regions resolve a thread plan; 40 unplanned by decision 9, none new; no escapes)Five mutations against the real gate, each caught by the intended branch — new unplanned region fails; a planned region that stops planning fails; a properly wired region notes and does not fail; an unbalanced BEGIN/END fails naming its own function; clean tree passes. The last is why
unbalancedis returned separately: the first attempt left the scanner inside a region for the rest of the file and printed 87 escape lines without ever saying the macro was unbalanced.The count is 40 and not 41 because
mixture_bindings.c's doc comment explains what holds across "thePy_BEGIN_ALLOW_THREADSregion", and a raw text search counted that sentence — the same mistakepublic_text_namesalready records, wherejet_sweepmatched a prose comment and was recorded as public API. Comments are now stripped, tracking/* */across lines.3. A forked worker says why it is about to hang
An OpenMP thread pool does not survive
fork: only the calling thread exists in the child, so the child waits at its first parallel computation with no exception, no traceback and no exit. The documentation said to use spawn and did not say what happens otherwise, so the advice was read after the hang.The child now warns, naming the hang and the remedy. The predicate is measured, not assumed: a fresh import reports 1 native thread, a model built under
set_threads(4)reports 4 — so it is off for a fork that is safe and on for one that is not. Read from/proc/self/status, because an OpenMP worker is not a Python thread andthreading.active_count()cannot see it.One of the four tests was not load-bearing until it was mutated:
test_a_fork_with_nothing_running_is_silentusedsimplefilter("error"), which the handler swallows on purpose, so it passed against a guard mutated to warn on every fork. It now asserts on the recorded warning list. Removing theos.register_at_forkcall is caught by the end-to-end fork test and by none of the other three, which is why that test exists.Verification
numeric(78 keys),math_values(58 quantities), saved-model sha256, thread independence at 1/4/16,exact(416 quantities).python_api.txtunchanged at 381 lines — every name added is underscore-private227a5fb3931208cfc4b6d9f91d43b2dd06dca6e915b171ecfa54f0c470d4f6b0, and 0 C files changed across all three commitsBaselines moved:
gil_regions.txtis new (69 lines);pytest_cases1238 → 1242, four ids added and none removed. Both documented inRERECORDED.mdnaming every line.4. Fitting gets a planned thread count, now that fitting is profiled
Decision 9 left fitting on the old behaviour "until it is profiled". Matched-sample sweep, five processes per thread count, median of medians, 16-core desktop at load 2.8–6.5:
Fitting parallelises weakly and degrades past 8. The behaviour decision 9 preserved asked
omp_get_max_threads()— 16 on this box, the worst point on both curves, and 2.2× worse than serial at order 2. Re-measured in auto mode after wiring: order 2 410.7 → 178.1 ms (2.31×), order 3 693.4 → 405.2 ms (1.71×), withparallel_plan()reporting 8.A fitting-specific ceiling of 4–6 was considered and rejected: ~7% / ~5% over the existing ceiling does not pay for a second measured constant that has to stay correct on machines nobody has measured.
The other 26 of the 40 are query-time calls decision 9's own rule already covered and Task 4 did not reach (mixture, query, publicness, scoring, distance, fingerprint, insights, sample, top_k). The analytics ceiling was measured on that class, so they need no separate profile.
Mechanical, and checked before any edit: all 40 regions sit in files that already include
python_bindings.h, none contains areturnorgoto, none lacks a matchingPy_END_ALLOW_THREADS. The replacement is comment-aware somixture_bindings.c's doc comment was not silently rewritten; it is corrected by hand, since the region it describes is no longer spelled that way.No recorded number moved across 40 entry points whose thread count changed — 78 numeric keys, 58 math values bit for bit, the saved-model sha256, thread independence at 1/4/16, and 416 quantities against enumeration all compared equal. That is the reduction design earning its keep: no floating-point sum reduction anywhere, and the sums that exist split into a fixed 256 chunks summed in index order. The extension hash did change (
227a5fb3…→4dc81355…), since this is the first commit in the series to edit C.The gate's line is now
OK thread plan wiring (all 69 GIL-releasing regions resolve a thread plan; no escapes). The baseline is kept although nothing is exempt any more: a region added later without the macro is a failure against it.5. Every region must NAME its thread count, not merely be planned
A review of commits 1-4 found the gate certifying something it does not check. It printed
OK thread plan wiring (all 69 ... resolve a thread plan)under the heading "The thread plan is actually in force". The macro scan proves a plan is resolved at the Python boundary; it cannot prove the region underneath names it. One did not:src/native/occupancy/poisson_binomial/pgf.ccarried nonum_threadsclause at all — the only such region insrc/native.py_publicness_pgfis one of the 40 wrapped in commit 4 and the wrap did nothing there, measured at 16 native threads afterset_threads(1).That is the same fault as the thirty drifting symbols and the 372-difference API check, reproduced while fixing them.
plan.hcarried the matching overstatement, untrue twice over: 29 regions name a budget-derived local rather thanbeam_threads(), and one named nothing.check_lib.thread_count_clausesis the check the policy plan specified as Task 5 and that was never built. Brace/paren-matched rather than regexed —conc ? (int)conc : 1contains parentheses, and a regex stopping at the first)reads it asconc ? (int. It fails on a region naming no count, and on a count outside_PERMITTED_THREAD_COUNTS— a tripwire in source, not a baseline, so adding a name is a review decision that forces the question "what resolves that count". Rejectsomp_get_max_threads(),omp_get_num_procs()and any literal.Both failure paths mutated against the real gate, plus 11 unit tests. New line:
OK thread counts (93 parallel regions, every one names a permitted thread count).Also from that review:
make test BUILD=elsewherelinked intoelsewhere/and then ranbuild/, reporting a pass for binaries it had not built; andbuild-tsan/was not ignored, somake tsandirtied the tree.6. The fork guard was mute exactly where warnings are loudest
Under
-W erroror a pytestfilterwarnings = error,warnings.warnraises — and theexcept Exception: passthat let a child which cannot warn still run caught it and threw it away. Measured: with warnings as errors, a child forked after a BEAM computation hung and printed nothing.Now has an
os.write(2, …)fallback of a constant built at import, degrading to a stderr line rather than an exception because an exception from an at-fork handler has nowhere to go. Test usescapfdnotcapsys(the write bypassessys.stderr) and fails when the fallback is removed.The message also stated a hang as fact on a deliberately broad predicate — a Jupyter kernel satisfies "more than one native thread" with BEAM never having run — so it now says may hang if this child uses BEAM. The docs sentence "A fork before any computation has run is safe, and is not warned about" was false for any multithreaded process and is replaced.
Not in this PR
make tsancannot run untilclang-19andlibomp-19-devare installedomp_get_num_procs, affinity, cgroup quota) is in logical CPUs, and the ceiling of 8 equals this box's physical core count — so on the machine it was measured on, "the algorithm stops scaling at 8" and "do not exceed the physical cores" are the same number. Deferred pending a second machinereturnhidden in a macro, abreak, areturnon theBEGINline itself,/*inside a string literal,PyEval_SaveThread, and a_C_FUNCregex that eats a name's first character when the return type is on its own line) — none occurs in the tree today, all would pass silently if introduced. Plus a branch-order gap in the wiring check that is unreachable at 0 unplanned, two unused locals, and the 40 moved ledger lines named by file rather than individuallyverification/tests(now 107 tests guarding the gate itself) is still not run by any CI workflow